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1.  SCOPE 


This  section  identifies  the  Ada  Compiler  Evaluation  CapabUtiy  (ACEC)  Release  3  0  produa. 
states  its  purpose,  and  summarkes  the  purpose  arid  ccxitents  of  this  Final  Technical  Repon 

/,/  IDENTIFICATION 

This  is  the  Finai  Technical  Report  for  the  ACEC  Release  3.0.  It  was  develcped  by  Boeing  De¬ 
fense  &  Spfce  Group.  Product  Support  Division  under  contract  to  the  Wright  Laboratory 

The  first  release  of  the  ACEC  program  developed  a  Software  Product  consisting  of  a  suite  of 
benchmark  lest  programs,  suppoit  tools,  a  Reader’s  Guide,  a  User's  Guide,  and  a  Venitw  De¬ 
scription  Document  (VDD). 

The  second  release  of  the  ACEC  corrected  problems  found  in  the  first  release;  approximately 
300  new  performance  tests,  assessors  for  program  library  systems,  symbolic  debuggen,  and 
system  diagnostics,  a  new  tool  to  simplify  the  preparation  of  input  to  MEDIAN,  and  a  Single 
System  Analysis  (SSA)  tool;  and  upgraded  supporting  documentation  to  permit  a  user  to  assess 
the  performance  of  Ada  compilation  systems. 

The  third  release  of  the  ACEC  significantly  improves  usability  and  coverage  It  adds; 

•  A  systematic  Pretest  phase  which  guid«i  the  user  through  the  adaptation  necessary  to  run  the 
ACEC  on  a  new  compilation  system  ar«l  tJ«5  recording  of  the  choices  mark  for  later  refer¬ 
ence. 

•  The  user  documenution  has  been  enhanced,  and  provides  more  information  to  simplify  exe¬ 
cution  of  the  ACEC  and  interpretation  of  the  results. 

•  The  sampte  command  files  have  been  modified  in  order  to  simplify  the  effon  required  to 
adapt  to  other  Ada  compilation  systems. 

•  New  performaxKe  tests  have  been  added,  including  lest  problems;  to  iktermine  the  order  of 
processing  of  alternatives  in  a  SELECT  Aatement;  to  determine  dv;  layout  of  arrays  (row  or 
column  major);  to  report  on  the  vari^ility  of  the  size  of  task  control  blocks,  activation  re¬ 
cords,  variant  records  and  objects  of  an  unconstrained  type;  to  calculate  task-switching  time: 
to  repon  whether  a  compiler  will  evaluate  an  arithmetic  expression  in  a  way  thtt  will  pro¬ 
duce  a  result  different  from  the  canonical  order;  to  provide  an  exsti^ie  application  of  an  in¬ 
ference  engine;  to  systematically  evaluate  the  variation  in  performance  as  the  size  of  a  sec¬ 
tion  of  code  increases:  to  compare  differences  in  coding  style  based  on  IF  statements  versus 
exceptions;  and  to  explore  the  nin-ume  overhead  of  entering  blocks  that  declare  uninitialized 
variables. 

•  The  ACEC  performance  tests  have  been  renamed,  repackaged  and  reorganized  into  groups 
and  subgroups  by  test  objective.  This  is  to  give  the  users  an  easier  conprehension  of  the  over 
1600  test  elements  and  to  provide  test  result  analysis  flexibility.  This  reorganization,  along 
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with  added  functionality,  allows  the  user  to  assign  weights  for  individual  tests  and  for  groups 
of  related  tests  for  analysis  purposes. 

•  The  daia-gathering  and  analysis  tasks  have  been  auionxated,  eliminaung  ntany  of  the  time- 
consuming  and  cumbersome  compile,  link  and  run  steps  in  previous  versions.  The  Compara¬ 
tive  Analysis  tool  provkles  confidence  intervals  on  system  factors  between  problems  wnhm 
groups  of  related  tests,  and  between  the  factors  calculated  for  different  groups. 

•  Capacity  tests  for  both  run  time  and  compile/Unk  time  have  been  added. 

•  Systematic  Compile  Speed  tests  have  been  added  These  tests  are  designed  to  test  the  sensi¬ 
tivity  of  compile  speeds  to  different  language  feature  usage. 

1.2  PURPOSE 

The  purpose  of  this  document  is  to  review  the  problems  encountered  and  the  lessons  learned  in 
the  process  of  developing  the  third  release  of  the  ACEC  Software  Product. 

The  descriptions  on  how  to  use  the  product  have  been  presented  in  the  Guides  Information  con¬ 
tained  in  the  Guides  will  not  be  repeated  here. 

The  numeric  results  of  the  ACEC  Release  3.0  testing  are  presented  in  the  ACEC  Software  Test 
Report  for  Release  3.0  and  will  not  be  repeated  here. 

1.3  INTRODUCTION 

The  ACEC  group  in  Boeing  Defense  &  Space  Group.  Product  Support  Division.  Avionics  Sys¬ 
tems  and  Software  Department,  tested  the  ACEC  Release  3.0  in  November  -  December  1991. 
The  ACEC  was  tested  on  five  trial  systems,  3  of  which  were  the  same  as  were  used  to  test 
ACEC  Release  2.0:  DEC  Ada  VAXA^MS  self-hosted;  TeleSoft  NAXJVMS  self-hosted;  and 
the  Verdix  scif-hosted  system  for  the  Silicon  Gri^hics  (a  UNIX-based  system).  The  VAXA'^S 
hosted  1750A  targeted  compilation  system  used  in  Release  2.0  was  replaced  with  the  TLD 
VAXA^MS  hosted  1750A  targeted  compilation  system  (using  a  different  target  processor).  The 
Meridian  self-hosted  compilation  system  for  the  DECStation  (another  UNIX-based  operating 
system)  was  used  in  place  of  the  ALSYS  self-hosted  compilation  system  on  the  Apollo. 

The  ACEC  Release  3.0  Software  Product  consists  of; 

•  A  set  of  tests  and  procedures  forming  a  Pretest  which  guides  an  ACEC  user  step-by-step 
through  the  process  of  adapting  the  ACEC  product  to  a  new  compilation  system. 

•  A  suite  of  performance  test  programs. 

•  A  set  of  supporting  packages. 

These  include  a  preprocessor  to  incorporate  the  timing  loop  in  the  rest  prograim,  a  portable 
implementation  of  a  math  library,  and  packages  defining  global  types,  variables,  and  subpro¬ 
grams  for  use  in  the  test  suite  and  analysis  tools. 
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•  A  set  of  analysis  tools. 

These  include  CONDENSE,  MENU,  Comparative  Analysis  (CA),  and  Single  System  Analy¬ 
sis  (SSA). 

•  A  set  of  assessors  for  diagnostics,  symbolic  debuggers,  program  library  systems,  and  compi¬ 
lation  system  capacity  limits. 

•  Sample  command  files  to  compile  and  execute  the  tools  and  tests  for  a  VMS  and  for  a 
UNIX-based  system  which  can  be  used  as  guides  for  porting  to  other  systents 

This  report  assumes  that  the  reader  is  familiar  with  the  ACEC  Release  3.0  User's  Guide,  the 
Reader’s  Guide,  and  the  Version  Description  Document  (VDD).  The  reader  should  refer  to  the 
VDD  Release  3.0  for  a  description  of  the  tests.  Information  from  the  Software  Test  Report  Re¬ 
lease  3.0  is  also  referenced. 

This  document  reviews  the  development  of  the  third  release  of  the  ACEC  Software  Product  It 
gives  an  analysis  of  results  and  a  review  of  the  lessons  learned. 
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2.  APPLICABLE  DOCUMENTS 

The  following  dociuncnts  axe  referenced  in  this  guide. 
2.1  GOVERNMElfT  DOCUMENTS 


MIL-STD-1815A  Reference  Manual  for  the  Ada  Programming  Language  (LRM) 
2 .2  NON -GOVERNMENT  DOCUMENTS 


D500- 1 2563-1 


D500- 12564-1 


D500- 12565-1 


D500-12570-1 


Ada  Compiler  Evaluation  Capability  (ACEC) 

Version  Description  Document  (VDD)  Release  3.0 

Boeing  Defense  &  Space  Group 

Produa  Support  Division 

P.O.  Box  7730 

Wichita,  Kansas 

Ada  Compiler  Evaluation  Capability  (ACEC) 
Technical  Operating  Report  (TOR) 

User’s  Guide  Release  3.0 
Boeing  Defense  &.  Space  Group 
Product  Support  Division 

Ada  Compiler  Evaluation  Capability  (ACEC) 
Technical  Operating  Report  (TOR) 

Reader’s  Guide  Release  3  0 
Boeing  Defense  &  Space  Group 
Produa  Support  Division 

Ada  Compiler  Evaluation  Capability  (ACEC) 
Software  Test  Report  Release  3.0 
Boeing  Defense  &  Space  Group 
Product  Support  Division 
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3.  ANALYSIS  OF  THE  TEST  SUITE 


The  following  table  displays  the  test  problems  where  one  of  the  trial  systems  used  in  FQT  a 
residual  for  execution  time  of  less  than  or  equal  to  a  tenth,  or  greater  than  or  equal  to  ten.  Such 
highly  variable  results  highlight  test  problems  where  systems  may  have  taken  very  different  ap¬ 
proaches.  and  may  identify  problems  where  systems  are  not  performing  comparable  operations. 
The  trial  systems  are  named  V_9l,  T_91,  S_91,  M_91,  and  D_91,  as  discussed  in  the  ACEC 
Software  Test  Repon  Release  3.0.  The  full  set  of  numeric  results  obtained  by  CA  on  the  trial 
system  data  are  presented  in  the  Software  Test  Report  Release  3.0  and  will  not  be  duplicated 
here. 

Some  test  problems  are  mentioned  more  than  once  because  they  had  exceptional  residuals  on 
more  than  one  compilation  system 


PROBLEM  NAME 

RESIDUAL 

io_sq_io_copy_02 

0.0 1 

dr_ba_bool_arrays_29 

0.0 1 

io,sq_io_copy_0 1 

0.0 1 

clr_aa_anray_aggreg_0 1 

O.Ol 

dr_ao_atTay_opcr_20 

O.Ol 

io_sq_io_scan_l  1 

0.01 

dr_aa_arTay_aggreg_0 1 

O.Ol 

dr_ba_bool_arrays_28 

0.01 

io_sq_io_copy_02 

0.02 

cir_ba_bool_array  s_  1 3 

0.02 

io_sq  Jo_copy„0 1 

0.02 

dr_ba_bool_array  s_  1 4 

0.02 

cir_ba_bool_arTays_l  5 

0.02 

dr  _ba_booI_array  s_  1 7 

0.02 

dr_ba_bool_anay  s_3  2 

0.02 

io_sc[_io_scan_  1 2 

0.02 

io_ds_io_04 

0.03 

dr_be_bool_express_  1 2 

0.03 

dr  _be_booi_express_  1 2a 

0.03 

dr_aa_array_aggrcg_0 1 

0.03 

io_tf_iext_io_flt_str_03 

0.03 

dr_ba_bool_arrays_29 

0.03 

dr_ba_bool_arrays_i  1 

0.03 

dr_aa_array_aggTeg_0 1 

0.03 

op_fa_aiias_08 

0.03 

dr_ba_bool_array  s_  1 7 

0.04 

dr  _ba_bool_arrays_l  2 

0.04 

io  _da_io_scan_08 

0.04 

op_fa_altas_12 

0.04 
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io_sq_io_l7 

io..tf_textjo_flt_str_01  0  04 

opjvjoop_invar_08  0.05 

io_tOext_io_flt_str_02  0.05 

do_rp_rcp_pack_24  0.05 

dr_ba_booLarrays_l5  0.05 

op_fs_foId_simp_24  0.05 

dr_ba_bool_arrays_28  0.05 

po_pa_djibrary_07  0.05 

io_dpjo_pattem_07  0.05 

do_rp_rep_pack_49  0.05 

do_rp_rep_pack_l9  0.05 

do_rp_rcp  _pack_54  0.05 

op_fs_fold_simp_l7  0.06 

io_du_io_unif_05  0.06 

dr_ba_bool_aiTays_32  0.06 

op_oe_order_of_eval_20  0.06 

io_da_io_scan_  15  0 .06 

dr_ba_bool„arrays_29  0.06 

io_dp_io_paitem_08  0.06 

dr_ba_bool_arrays_14  0.06 

dr_tc_type_cnum_08  0.06 

dr_ba_bool_arrays_23  0.06 

op_fs_fold_simp_21  0.06 

op_Iv_loop_invar_09  0.06 

io_ti_text_io_int_str_02  0.06 

io_dajo_scan_04  0.06 

iojf_text_io_flt_str_Ol  0.06 

io_tf_tcxt_io_flt_str_03  0.07 

io_dp_io_pattem_06  0.07 

ms_ec_express_cat_01  0.07 

io_dr_io_rccur_0l  0.07 

io_di_io_80_20_03  0.07 

io_dp_io_pattcm_07  0.08 

io_tx_io_0l  0.08 

dr_ba_bool_airays_13  0.08 

dr_ba_booLaiTays_14  0.08 

xh_er  cxcept_raise_03  0.08 

io_dijo_80_20_05  0.08 

opjv_loop_invar_01  0.08 

io_drjo_recur_01  0.08 

to  dp_io jpattem_08  0.09 

jo_txjo_09  0.09 

op_as_algc_simp„06  0.09 

io_tx_io_24  0.09 

dr_ba_bool_arrays_15  0.09 


do_cu_conv_unck_0 1  0 .09 

nis_ec_express_cat_Ol  0.09 

<ir_ao_anay_oper_33  0.10 

dr„b€_bool_express_  1 2a  0. 1 0 

st_is_if „code_sty  le_29  0.10 

dr_ao_array_oper_32  0.10 

dr_ba_bool_an'ays_05  0 . 1 0 

dr_ba_bool_arrays_07  0.10 

dr_ba_bool_arrays_  11  0.10 

dr_rd_rec_discr_02  0.10 

st_is_if_code_style_29  1 1 .34 

io_tx_io_09  12.17 

io_tx_inst_05  12.41 

io_tx_inst_03  13.54 

io_tx_inst_02  19.16 

io_tx_inst_01  19.20 

io  tx  inst  04  24.12 


The  following  alphabetic  list  of  performance  problem  names  describes  the  features  in  each  of 
these  problems  and  the  systems  which  were  flagged  as  exceptional 


PROBLEM  NAME  APPARENT  REASON 


do_cu_con v_unck_0 1 


do_rp_rep_pack_  1 9 


do_ip_rep_pack_24 


do_rp_rcp_pack_49 


This  problem  tests  unchecked  conversions 
by  passing  a  (converted)  array  parameter 
to  a  function.  No  explicit  code  is 
required  to  support  the  conversion. 

V_91  was  exceptionally  slow  and  S_9l  was 
exceptionally  fast.  The  low  subprogram 
overhead  on  S_91  contributed  to  its 
high  performance  on  this  problem. 

This  is  a  test  of  fetching  elements  from 
a  packed  array.  M_91  was  exceptionally 
fast  on  this  problem,  perhaps  because 
the  system  did  NOT  pack  the  array, 
permitting  faster  access  than  the  other  systems. 
This  is  a  test  of  fetching  elements  from 
a  packed  array.  M_91  was  exceptionally 
fast  on  this  problem,  perht^s  because 
the  system  did  NOT  pack  the  array, 
permitting  faster  ac  ^ss  than  the  other  systems. 
This  is  a  test  of  referencing  elements 
from  packed  and  unpacked  arrays.  M_9l 
was  exceptionally  fast  on  this  problem, 
perhaps  because  the  system  did  NOT 
pack  the  array,  permitting  faster 
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do_rp_rcp_pack_54 


dr_aa_array_aggreg_0 1 


dr_ao_aiTay_oper_20 


dr_ao_array_oper_32 


dr  _ao_array_oper_3  3 


dr_ba_bool_arrays_05 


dr_ba_bool_arrays_07 


dr_ba_bool_arrays_l  1 


dr_ba_bool_aiTays_  1 2 


access  than  the  other  systems. 

This  is  a  test  of  fetching  elements  from 
a  packed  array.  M_91  was  exceptionally 
fast  on  this  problem,  perhaps  because 
the  system  did  NOT  pack  the  array, 
permitting  faster  access  than  the  other  systems. 
This  problem  declares  a  constant  array 
initialized  with  literals.  M_9l  docs 
it  particularly  poorly  and  V_91 ,  T_91 , 

S_91  and  D_91  (by  comparison  to  the 

mean  time  for  the  problem)  do  it  panicuiarly  well. 

This  problem  deals  with  initializing 

a  constant  array  with  another  array 

(which  naight  be  optimized  by  effectively 

renaming).  D_9l  was  exceptionally  fast 

ana  M_91  was  exceptionally  slow  on  this  problem. 

This  problem  makes  repeated  accesses 

to  an  element  of  a  2-D  array  whose 

actual  subscripts  are  nested  FOR  loop 

indexes.  This  is  an  important 

special  case  of  strength  reduction. 

M_9l  did  exceptionally  well. 

This  problem  makes  repeated  accesses 
to  an  element  of  a  2-D  array  whose 
actual  subscripts  are  nested  FOR  loop 
indexes.  This  is  an  important 
special  case  of  strength  reduction. 

M_91  did  exceptionally  well. 

This  problem  tests  boolean  OR  operation 
on  a  small  unpacked  boolean  array. 

V_9l  was  exceptionally  fast.  Apparently 
both  it  and  T_91  did  NOT  call  on 
run-time  library  routines  to  perform  the  work. 

This  problem  tests  boolean  XOR  operation 
on  a  small  unpacked  boolean  array. 

V„9l  was  exceptionally  fast.  Apparently 
both  it  and  T_91  did  NOT  call  on 
run-time  library  routines  to  perform  the  work. 

This  problem  performs  boolean 
operations  on  small  packed  boolean 
arrays.  T_91  and  S_91  were  both 
exceptionally  fast  (only  a  small  number 
of  instmetions  were  performed). 

This  problem  performs  boolean 
operations  on  small  packed  boolean 
arrays.  T_Vi  was  exceptionally  fast 
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cir_baJxK)l_arrays_  1 3 


dr  _ba  _boo  i  _arTay  s  _  1 4 


dr_ba_bool_arrays_l  5 


dr  _ba  .bool  .array  s  .  1 7 


dr_ba_bool  _anays.23 


cir_ba_booLarrays_28 


dr  _ba_bool  .arrays  _  29 


dr_ba_bool_arTays_32 


requuing  only  a  small  number  of 
uuouctions  lo  be  cxecmed. 

This  problem  performs  boolean 
operations  oti  small  packed  boolean 
arrays.  T_9l  and  S_9S  were 
exceptionally  fast  requiring  cmly  a 
small  number  of  mstrucuons  lo  be 
executed  M_9l  was  exceptionally  slow 
This  problem  performs  a  simple  boolean 
AND  on  two  small  packed  boolean 
arrays  V.91.  T_9l  ami  S_9I  were 
exceptionally  fausi  requiring  only  a 
small  number  of  mstrucuons  to  be  executed 
This  problem  performs  a  simple  boolean 
OR  on  two  small  packed  boolean 
arrays.  V_9l,T_91  arKlS_9l  were 
exceptionally  fast  requiring  only  a 
small  number  of  mstrucuons  to  be  executed 
This  problem  performs  a  simple  boolean 
XOR  on  two  small  packed  boolean 
arrays.  V_91  and  T_9l  were 
exceptionally  fast  requiring  only  a 
small  number  of  mstrucuons  to  be  executed 
This  problem  performs  simple  boolean 
AND  ar«l  "=  ’  operations  on  a  long 
unpacked  boolean  array  T_9l  supports 
It  exceptionally  well,  fierhaps  with  mime  code 
This  problem  performs  an  OR  operauon 
on  a  packed  boolean  array  of  size  16 
V.91  andT_9l  were  exceptionally  fast 
(and  S_91  was  also  fsuriy  fast),  each 
taking  a  few  mstrucuons  to  perfonm 
the  test  problem. 

This  problem  is  similar  to 
dr_ba_bool_airays_28  but  uses  for  one 
operand  a  named  aggregate  with  literal 
values  rather  than  a  sumple  variable. 

For  all  but  M_9l.  the  times  of  the 
problems  aie  indistinguishable  (as 
was  hoped).  V_9l.  T_9l  and  S.91  were 
exceptiofiaily  fast,  and  M_91  was 
exceptionally  slow  (apparently 
constnicting  the  aggregate  at  run  time) 

This  problem  is  similar  to 
dr_ba.bool_arrays_28  but  uses  for  one 
operand  a  subscripted  array  refcience. 


13 


dr_be_bool  _express  _  1 2 
dr  _be  _bool  _ex  press  _  1 2  a 


dr_rd_rcc_disGr_02 


dr_te_type_enum_08 


10  da  lo  scan_04 


io_da_io_scan_08 
io_da_io_scan_  1 5 
io_di_io_80_20_03 

io_di_io_80_20_05 

io_dp_io_pattem_06 

io_dp_io_pattem_07 

io_dp_io_pattem_08 

io_dr_io_rccur_0 1 

io„ds_io_04 

io_du_io_unif_05 


rather  than  a  simple  variable. 

V_9l  and  T_91  wenr  exceptionally  fast, 
and  S_9l  was  only  slightly  slower 
This  problem  performs  bit  mampulation 
using  array  indexes  on  a  packed  array 
T_91  IS  exceptionally  fast. 

This  problem  performs  bit  mampulation  using 
array-wide  logical  operators  on  a 
packed  array.  T_91  and  S_9l  are 
exceptionally  fast. 

This  problem  checks  a  discnmmated 
record  and  raises  CONSTRAINT.ERROR 
D_9l  was  exceptionally  fast  at 
processing  the  exception.  T_9l  was 
almost  exceptional. 

This  test  problem  explores  using 
representation  clauses  to  specify 
values  for  an  enumerated  type.  D_91  was 
exceptionally  fast. 

This  is  an  I/O  opieration  where  large 
differences  between  systems  are 
expected  and  where  caching  and  buffering 
and  general  Curating  System  I/O 
processing  greatly  influences 
performance.  M_91  was  exceptionally  fast. 
Direct  flic  I/O  operation  test.  M_91 
was  exceptionally  fast. 

Direct  file  I/O  operation  test.  S_91 
was  exceptionally  fast. 

Direct  file  I/O  operation  test  on  a 
file  with  10_000  records.  M_91  was 
exceptionally  fast. 

Dirert  file  I/O  operation  test  on  a 
file  with  100_000  records.  M_91  was 
exceptionally  fast. 

Direct  file  I/O  operation  test. 

S_9l  and  M_91  were  exceptionally  fast. 
DircCT  file  I/O  operation  test. 

S_9l  and  M_91  were  exceptionally  fast. 
Direct  file  I/O  operation  test. 

S_91  and  M_91  were  exceptionally  fast. 
Direct  file  I/O  operation  test. 

S_9l  smd  M_91  were  exceptionally  fast. 

I/O  operation.  M_91  was  exceptionally  fast. 
Direct  file  I/O  operation  test  reading 
from  file  with  100_000  records.  S_91 
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io_sq_io_l7 

was  exceptionaUy  fast. 

Open  and  close  a  sequential  file.  M_9l 

io_s<l_io_copy_0 1 

was  exceptionally  fast. 

Sequential  file  I/O  operation  test. 

io_sq_io_copy_02 

V_91  and  M_91  were  exceptionally  fast. 
Sequential  file  I/O  operation  test. 

io_sq_io_scan_l  1 

V_9l  andM_91  were  exceptionally  fast. 
Sequential  I/O  operation  test,  writing 

io_sq_io_scan_12 

variable  length  records.  V_91  was 
exceptionally  fast. 

Sequential  I/O  operation  test,  reading 

io_tf_tcxt_io_flt_str_0 1 

variable  length  records.  V_91  was 
exceptionally  fast. 

Put  to  a  string.  Test  of  FLOAT_IO 

io_tf_text_io_flt_str_02 

conversions.  V_91  and  M_91  were 
exceptionally  fast. 

Put  to  a  string.  Test  of  FLOAT_IO 

io_tf_text_io_flt_str_03 

conversions.  V_91  was  exceptionally  fast. 
Put  to  a  string.  Test  of  FLOAT_IO 

io_ti_text_io_int_str_02 

conversions.  V_9I  and  M_9l  were 
exceptionally  fast. 

Put  to  a  string.  Test  of  INTEGER_IO 

io_tx_inst_01 

conversions.  V_91  was  exceptionally  fast. 
Instantiate  enumeration  I/O.  D_9l  was 

io_tx_inst_02 

exceptionally  slow. 

Instantiate  enumeration  I/O.  D_91  was 

io_tx_inst_03 

exceptionally  slow. 

String  assignment.  D_91  was 

io_tx_inst_04 

exceptionally  slow. 

Use  of  ’image  attribute.  D_91  was 

io_tx_inst_05 

exceptionally  slow. 

Literal  string  assignment.  D_91  was 

io_tx_io_01 

exceptionally  slow. 

Open  and  close  a  text  file.  M_9I  was 

io_tx_io_09 

exceptionally  fast. 

I/O  operation,  reset  file.  S_9I  was 

io_tx_io_24 

exceptionally  slow.  M_91  was 
exceptionally  fast. 

Console  I/O  operation.  V_91  was 

ms_ec_exprcss_cat_0 1 

exceptionaUy  fast. 

This  problem  uses  the  string  concatenation 

op_as_aigc_simp_06 

operations  and  slices.  V_91  and  T_91 
were  exceptionally  fast. 

The  problem  contains  a  multiple  by  a 

literal  ”1."  It  was  exceptionaUy  fast 
on  V_9l,  because  it  folded  the  operation. 
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op_fa_alias_08 
op_fa_alias_l2 
op_fs_fold_simp_  i  7 

op_fs_fold_simp_2 1 


op_fs_fold_simp_24 
op_l  v_loop_invar_0 1 


op_Iv_loop_invar_08 

op_l''_loop_invar_09 

op_oc_order_of_eval_20 


po_pa_d_Ubrary_07 


st_is_if_code_style_29 

xh_er_except_raise_03 


This  problem  contains  complex  foldable 
code.  T_91  was  exceptionally  fast. 

This  problem  contains  complex  foldable 
code.  T_9l  was  exceptionally  fast. 

This  problem  contains  code  with  foldable 
subexpressions.  S_91  was  exceptionally 
fast,  and  T_91  was  also  fairly  fast. 

This  problem  contains’ code  with  foldable 
subexpressions,  if  a  compiler  propagates 
a  literal  assignment  to  a  variable 
declared  with  a  CONSTANT  attnbute. 

S_91  was  exceptionally  fast. 

This  problem  contains  foldable  literal 
subexpressions.  S_91  was  exceptionally  fast. 
This  problem  contains  a  FOR  loop  with  one 
invariant  assignment  within  it.  It 
could  be  optimized  into  one  assignment, 
removing  the  invariant  code  from  the 
loop  and  removing  the  (now)  redundant 
loop  overhead.  M_9l  was  exceptionally  fast. 
This  problem  tests  several  cases  where 
applying  loop  invariant  motion  could  be 
applied  to  reordering  expressions 
across  parentheses.  V_91  was  exceptionally  fast. 
This  is  a  version  of  op_lv_loop_invar_08 
using  type  FLOAT9. 

One  of  the  systems  uses  extended 
precision  temporaries  to  evaluate  an 
expression  and  in  so  doing  avoids 
raising  an  exception  (exception  raising 
can  be  expensive).  This  is  a  valid 
comparison  test,  even  though  the 
problem  will  raise  an  exception  on  some 
systems  and  not  on  others  because  the 
intent  of  the  test  is  to  determine 
precisely  this  behavior.  S_9l  was 
exceptionally  fast. 

This  problem  tests  for  elaborating  a 
package  declaration  which  requites 
dynamically-sized  storage.  M_91  was 
exceptionally  fast. 

This  problem  raises  a  CONSTRAINT_ERROR 
exception  when  converting  to  a  user- 
defined  subtype.  T_9l  was  exceptionally  fast. 
This  problem  declares  a  block  with 
an  exception  handler  for  a  user- 
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defined  exception.  The  exception 
is  NOT  raised,  so  the  problem  is  a 
measure  of  block  entry  and  exit  time. 

S_91  was  exceptionally  fast  (and 
V_91  was  almost  cxcefKional. 

In  this  list  of  exceptional  test  problems,  I/O  operations  occur  frequently,  followed  by  operations 
on  packed  boolean  arrays  and  (to  a  lesser  extent)  exception  processing.  Examitung  the  particu¬ 
lar  systems  marked  as  exceptional  (sec  Test  Repon  for  details)  shows  that  h  is  not  true  that 
compilation  systems  follow  a  simple  linear  ranking,  with  all  systems  optimizing  the  "simple ' 
problems  and  the  better  systems  optimizing  more  of  the  '  harder  '  problems.  A  system  which 
does  not  generally  optimize  many  problems  is  sometimes  the  only  system  to  optimize  one  par¬ 
ticular  problem.  This  behavior  may  not  match  a  priori  expectations,  but  it  does  reflect  actual 
behavior. 

3.1  CONFIRMED  EXPECTATIONS 

The  following  subsections  list  design  decisions  which  worked  well. 

3.1.1  C omparative  Analysis  I CA } 

In  analyzing  results  from  several  systems,  some  type  of  statistical  analysis  tool  such  as  CA  is  a 
practical  necessity  for  a  test  suite  with  more  than  a  small  number  of  test  problems.  The  analysis 
focuses  the  anention  of  the  ACEC  users  on  the  test  problems  with  "unusual"  results.  It  pennits  a 
form  of  report-by-exception  where  ACEC  users  can  concentrate  their  anention  on  the  test  prob¬ 
lems  with  anomalous  performance.  Since  most  test  problems  will  not  be  flagged  by  CA  as 
outliers,  ACEC  users  will  be  able  to  "skim"  over  most  of  them  and  focus  on  those  where  large 
differences  between  systems  were  observed. 

Without  any  comparative  analysis  tool,  a  test  suite  would  require  users  to  "understand"  each  of 
the  test  problems,  at  least  to  know  if  a  result  on  one  system  was  good.  bad.  or  indifferent.  The 
residual  matrix  is  very  helpful  since  it  establishes  a  normalized  metric  for  test  results  where  a 
residual  close  to  one  is  "typical."  No  aiudysis  tool  can  extraa  more  information  from  a  set  of 
data  than  is  implicit  in  the  collection  of  "raw  ’  measurements,  but  it  can  make  the  relationships 
between  data  more  apparent.  It  would  be  very  easy  to  overlook  the  fact  that  a  system  executes 
some  test  problems  twice  as  fast  as  typical  when  all  the  data  is  presented  as  one  large  table  of 
timing  measurements.  It  is  important  to  prevent  users  from  being  overwhelmed  by  the  volume 
of  data  and  CA  serves  this  role. 

Changes  added  to  CA  in  Release  3.0  provided  features  which  should  be  useful: 

•  Analysis  by  groups 

The  CA  tool  now  performs  its  analysis  on  groups  of  related  problems  rather  than  on  the  en¬ 
tire  set  of  test  problems,  and  does  a  summary  analysis  over  the  groups.  This  will  permit  users 
to  more  easily  discern  the  relative  performance  of  systems  on  the  (preselected)  groups.  For 
example,  comparing  system  performance  on  I/O-intensive  test  problems  is  more  straightfor- 
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ward  now  than  in  Release  2.0  because  Release  3.0  has  identified  a  set  of  I/O  test  problems 
and  CA  will  compare  these  sets  of  problems  between  systems. 

•  Confidence  intervals 

The  CA  tool  now  calculates  confidence  intervals  rather  than  point  estimates  for  system  fac¬ 
tors,  and  will  identify  whether  differences  in  system  factors  found  by  CA  are  statistically  sig¬ 
nificant.  Using  the  Release  2.0  MEDIAN  tool,  some  users  may  have  concluded  that  one  sys¬ 
tem  was  better  than  another  because  the  system  factor  was  bencr,  when  in  fact  the  difference 
between  the  systems  was  not  statistically  significant.  This  was  the  major  reason  for  the 
change  in  the  statistical  analysis  algorithm. 

•  Initial  report  summary 

Several  changes  were  made  to  the  CA  reports  to  help  users.  Provisions  were  made  for  users 
to  provide  descriptive  text  about  the  systems  being  compared,  in  order  to  more  clearly  iden¬ 
tify  the  system  on  the  CA  report.  Users  are  encouraged  to  provide  version  numbers  of  com¬ 
pilers  and  operating  systems  and  to  describe  the  hardware  configuration  tested.  System  fac¬ 
tors  and  confidence  intervals  are  displayed  graphically.  Graphics  also  show  whether  there  is 
a  statistically  significant  difference  between  computed  system  factors. 

CA  shows  the  count  of  errors  of  each  type  on  each  system.  Error  data  is  important  because  a 
fast  system  which  doesn't  support  the  full  Ada  language  is  not  necessarily  "better  ”  than  a 
slower  system  which  supports  more  of  the  language. 

•  Weights 

CA  now  provides  the  capability  for  users  to  provide  weights  for  individual  groups,  or  for  ui- 
dividuai  test  problems. 

•  Performance 

It  is  now  considerably  faster  to  rc*analyze  sets  of  data,  including  and  excluding  different  sys¬ 
tems.  in  CA  than  it  was  with  MEDIAN  in  Release  2.0.  In  Release  3.0.  it  is  not  necessary  to 
invoke  the  Ada  compiler  to  process  the  performance  data.  This  enhanced  performance  in 
reporting  should  encourage  more  exploration  of  the  data  that  the  users  collect. 

The  analysis  by  groups  and  the  confidence  levels  fill  a  need  reported  by  a  number  of  ACEC 
users. 

3.1.2  Single  System  Analysis 

The  Single  System  Analysis  (SSA)  tool  was  not  substantially  changed  firom  Release  2.0  (other 
than  reflecting  the  renaming  of  test  problems).  It  still  provides  an  automated  way  of  comparing 
and  displaying  results  from  sets  of  related  performance  test  problons.  ACEC  users  had  to 
manually  compare  results  of  related  tests  before  the  deveiofnnent  of  this  tool,  however,  the 
speed  and  automated  nature  of  the  tool  make  such  comparisons  much  simpler. 
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J.I  J  Include 


The  Include  tool  (zg_incld)  is  an  ACEC  support  tool  used  to  tcxmally  expand  Ada  source  text 
It  is  used  to  insert  the  timing  loop  code  into  the  test  programs.  Refer  to  the  User’s  Guide  for  a 
discussion  on  the  use  of  this  tool. 

It  permits  modification  of  the  timing  code: 

•  To  accommodate  measuring  either  Central  Processing  Unit  (CPU)  time  or  elapsed  tune 

•  To  accommodate  the  lack  of  support  for  the  label’ADDRESS  attribute. 

The  flexibility  provided  by  the  Include  processing  has  proved  valuable  in  Release  3.0  because 
significant  changes  were  made  to  the  timing  loop  (refer  to  the  next  section)  by  changing  a  few 
files.  It  was  not  necessary  to  modify  the  source  code  bracketing  all  1627  test  problems 

3.1.4  Timing 

The  timing  loop  code  is  responsible  for  measuring  the  execution  time  and  code  expansion  for 
each  test  problem.  ACEC  users  wishing  to  construct  their  own  test  problems  will  find  the  con¬ 
ventions  for  using  the  timing  loop  and  how  to  isolate  language  features  to  be  tested  are  dis¬ 
cussed  in  the  Reader's  Guide. 

The  timing  loop  code  was  changed  from  the  second  release  in  several  ways: 

•  INITTIME 

The  approach  to  initializing  timing  loop  parameters  was  changed.  In  Release  2.0,  the  initiali¬ 
zation  code  was  inserted  by  Include  into  every  test  program  and  as  a  result  was  compiled  and 
executed  many  times  (once  for  each  test  program).  In  Release  3.0,  the  initialization  code  is 
compiled  and  executed  once  for  each  compilation  option  (optimization,  checking,  no- 
optimization,  and  checking  suppressed).  The  initialization  code  in  each  test  program  copies 
the  timing  loop  parameters  from  files  created  during  Pretest  (see  section  3.1.9  for  a  discus¬ 
sion  of  the  ACEC  Pretest)  and  then  inserts  a  section  of  code  which  will  measure  the  time  to 
execute  a  sample  "null"  loop  and  verify  that  the  copied  timing  loop  parameters  are  consistent 
with  the  measurements  of  this  sample  loop.  If  this  verification  checking  were  not  performed, 
wrong  timings  could  be  obtained  because  different  compilation  options  (suppression  of 
checking,  optimization  levels,  etc.)  can  change  the  values  of  the  timing  loop  parameters. 

•  This  approach  reduces  the  total  lines  of  code  passed  through  the  compiler  (the  verification 
code  is  much  smaller  than  the  initialization  code  was  in  Release  2.0).  The  amount  of  time 
spent  executing  initialization  code  in  each  test  program  is  also  reduced  because  one  verifica¬ 
tion  loop  can  execute  much  faster  than  the  "full"  initialization  code. 


An  option  is  provided  to  use  zg.incld  to  incorporate  the  "friU"  initialization  code,  as  was 
done  in  Release  2.0,  for  users  who  wish  to  explore  the  performance  effects  of  systems  with 
many  compiler  options. 
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•  EXCESSIVE_TIME 


Ail  the  test  programs  in  the  Storage_Reclamation_Implicit  subgroup  in  Release  3.0  make  an 
initial  measurement  to  estimate  the  time  to  run  the  problem  to  completion  (the  code  is  exe¬ 
cuted  100_0(K)  times).  If  this  estimate  is  larger  than  the  value  EXCESSrVE_'nME,  the  test 
problem  execution  is  skipped  and  a  special  error  code  written. 

In  Release  2.0,  only  seven  of  the  problems  observed  to  exceed  time  Limits  on  the  trial  sys¬ 
tems  included  this  check.  The  Release  3.0  approach  will  be  safer  in  skipping  long  running 
test  problems  unless  the  user  explicitly  modifies  code  to  permit  longer  running  problems  to 
complete. 

•  EXCESSIVE^VARIABILITY 

In  order  to  reduce  the  number  of  test  problems  which  get  unreliable  measurement  error 
codes,  the  value  of  EXCESSrVE_VARIABILITY  was  reduced  from  2.0  to  1.25.  This  will 
have  the  effect  of  making  it  more  likely  that  the  timing  loop  control  logic  will  detect  exces¬ 
sive  variability  and  "jump"  to  a  larger  value  of  the  inner  loop  count,  producing  fewer  urueli- 
able  measurements. 

A  reduction  in  the  unreliable  measurement  error  codes  was  observed  on  the  trial  systems. 

•  ERR.LARGE.NEGATTVE.TIME 

The  timing  loop  code  uses  a  dual-loop  ^iproach  which  calculates  the  execution  time  for  a 
test  problem  by  subtracting  the  null  loop  time  from  the  elapsed  time  per  iterations.  When  the 
elapsed  time  per  iteration  is  less  than  the  null  loop  time,  a  spurious  negative  time  measure¬ 
ment  might  be  indicated.  When  this  happened  in  release  2.0,  a  zero  time  was  reported  when 
the  negative  time  was  less  than  the  variation  in  the  null  loop  time  and  an  unreliable  time  was 
reported  otherwise.  This  was  changed  in  release  3.0  so  that  a  new  error  code 
(ERR_LARGE._NEGAnVE_TIME)  is  returned  when  the  elapsed  time  per  iteration  is  sig¬ 
nificantly  less  than  the  null  loop  time. 

The  changes  made  in  Release  3.0  will  both  enhance  the  efficiency  of  gathering  data  {it  should 
take  less  time  to  collect  the  measurements)  and  increase  the  chances  that  reliable  data  will  be 
collected. 

The  compile  and  link  times  in  Release  3.0  are  measured  and  output  by  Ada  programs  rather 
than  job  control  procedures.  This  peimits  easier  adaptation  and  provides  for  automated  collec¬ 
tion  of  data  for  analysis  because  the  format  of  the  compilation/Iink  time  data  is  implementation- 
independent  (it  is  output  as  a  value  of  the  predefined  time  DURATION). 

3.1.5  Menu  Interface  For  Analysis  Tools 

The  new  interactive  MENU  interface  to  the  analysis  tools  provided  in  Release  3.0  provides 
benefits  in  several  areas: 
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•  It  is  easier  to  use. 


The  MENU  interface  to  the  analysis  tools  is  easier  to  use  and  faster  to  learn  than  the  manual 
manipulations  which  would  have  been  required  to  perfonn  the  corresponding  options  m  Re¬ 
lease  2.0. 

The  MENU  interface  should  increase  the  confidence  of  new  ACEC  users  who  will  be  able  to 
obtain  useful  results  with  less  effon  than  with  pnor  releases. 

•  It  is  faster. 

The  CA  tool  now  runs  much  faster  because  a  new  analysis  wUl  not  involve  a  recompilation 
and  link.  The  faster  execution  time  of  the  tool  will  make  interactive  use  feasible  and  encour¬ 
age  ACEC  users  to  explore  more  analysis  options. 

•  It  eliminates  some  misleading  results. 

The  CONDENSE  tool  (comparable  to  Release  2.0  FORMAT  tool)  reads  logs  and  database 
files  and  produces  report  files  showing  missing  data,  erroneous  problems  and  duplicate  prob¬ 
lems.  It  includes  consistency  checking  to  avoid  some  tyjjes  of  errors.  For  example,  in  re¬ 
lease  2.0  the  compilation  time  was  always  reported.  A  compiler  might  fail  quickly  and  reject 
a  valid  ACEC  program.  This  behavior  distoned  the  analysis  of  compilation  speeds  because 
it  made  a  system  which  failed  quickly  look  fast  when  analyzing  compilation  times.  In  re¬ 
lease  3.0,  the  compilation  time  for  a  program  is  set  to  an  error  code  when  there  is  no  valid 
execution  time  for  the  test  problems  contained  within  it. 

3.1.6  Guides 

The  writing  of  effective  user  documentation  is  difficult  and  time  consuming.  It  is  also  very  im- 
piortant  to  the  usability  of  a  product,  panicularly  when  there  is  no  provision  for  telephone  "hot¬ 
lines"  for  helping  users  who  run  into  problems. 

The  ACEC  user  documentation  is  extensive. 

The  User’s  Guide  presents  a  step-by-step  guide  to  using  the  ACEC.  It  now  includes  a 
Troubleshooting  Guide  appendix  discussing  common  problems,  symptoms  and  workarounds.  It 
includes  copies  of  all  the  assessor  stunmary  report  forms. 

The  Reader’s  Guide  discusses  issues  involved  in  understanding  the  results  produced  by  the  test 
suite,  discusses  background  issues  in  evaluation  and  benchmarking,  and  provides  guidelines  for 
users  writing  their  own  test  problems. 

The  ACEC  Version  Description  Document  (VDD)  contains  useful  information  about  the  test 
suite  itself,  (Ascriptions  of  the  individual  test  problems  and  descriptions  of  the  assessor  scenar¬ 
ios. 

The  revised  documents  were  well  received  by  the  beta  test  sites. 
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3.17  Assessors 


Release  3.0  of  the  ACEC  includes  assessors  for  the  Ada  program  library  management  system, 
the  diagnostic  messages,  the  symbolic  debugger  and  system  capacities. 

The  Capacity  Assessor  is  new  in  Release  3.0  and  contains  32  compile-time  and  9  run-time  fea¬ 
ture  tests.  The  Capacity  Assessor  requires  some  system-dependent  adsqjtation  in  the  host  system 
command  files  which  was  not  particularly  difficult  on  any  of  the  trial  systems.  When  runnuig 
the  edacity  Assessor  with  limits  large  enough  to  find  maximum  system  capacities,  the  tests 
can  take  a  long  time  to  complete;  however,  this  phenomena  is  expected  on  systems  which  have 
large  limits. 

The  assessors  have  generally  worked  well,  however,  they  do  require  more  programmer  effon 
than  we  might  have  liked  because  the  ACEC  user  must  manually  adapt  them  to  each  system 
being  evaluated. 

3.1.8  Renaming 

All  the  test  problems  and  tools  have  been  renamed  and  repackaged  to  reflect  logical  groupings 
of  related  programs. 

The  performance  tests  have  been  organized  into  groups  and  subgroups  (reflected  in  the  first  five 
characters  of  the  file  name)  making  it  easy  to  select  subsets  of  test  problems  to  run. 

In  general,  each  performance  test  is  now  in  a  separate  compilation  unit.  Perfoimancc  test  pro¬ 
grams  WITH  a  group  of  procedures  containing  individual  tests  and  then  call  on  them.  Before 
compiling  the  test  problem  procedures,  a  file  with  the  same  name  is  compiled  which  will  write 
an  enor  code  indicating  compile-time  failure  if  it  is  called.  If  there  is  an  error  in  compiling  ;he 
"real"  problem,  the  "dummy"  version  which  writes  an  error  code  will  be  linked  into  the  test  pro¬ 
gram.  This  technique  has  worked  out  well  in  permitting  automated  processing  of  many  prob¬ 
lems  with  compUe-time  errors.  It  also  reduces  the  number  of  packaging  errors. 

3.1.9  Pretest 

A  Pretest  section  has  been  added  which  leads  the  user  step-by-step  through  the  process  of 
adapting  the  ACEC  to  a  new  compilation  system  and  verifying  that  needed  facilities  are  pre¬ 
sent.  It  checks  the  acctuacy  of  the  system  clock,  the  math  library,  determines  timing  loop  pa¬ 
rameters,  checks  if  label’ ADDRESS  is  supported,  and  compiles  the  baseline  files,  the  programs 
which  write  compiler/linker  time  stamps,  and  the  analysis  tools. 

Beta  testers  reported  they  particularly  liked  this  addition.  It  can  reduce  anxiety  of  users  adapt¬ 
ing  the  ACEC  by  helping  them  avoid  blind  alleys  in  adapting  the  ACEC  to  a  new  compilation 
system.  The  summary  report  form  provides  a  document  for  recording  the  changes  made  in 
adapting  the  ACEC  and  remembering  the  configuration  tested.  One  user  of  an  earlier  version  of 
the  ACEC  reported  that  he  didn’t  find  out  that  the  analysis  tools  would  not  cori^ile  and  nm  on 
the  system  he  was  evaluating  until  he  had  run  all  the  performance  tests  and  did  not  have  enough 
time  left  on  his  evaluation  schedule  to  develop  an  alternative.  With  release  3.0  he  would  have 
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known  about  the  problem  earlier  and  might  have  developed  alternatives  such  as  genmg  a  cor¬ 
rected  version  from  the  implementor  of  the  compiler,  modified  code  in  the  analysis  tools  to 
work  around  problems  in  the  compiler,  or  arranged  to  use  a  different  compilation  system  to  per¬ 
form  the  analysis. 

i.2  PROBLEMS  ENCOUNTERED 

During  the  development  of  the  ACEC,  the  test  suite  and  tools  were  executed  on  multiple  sys¬ 
tems  both  to  verify  portability  of  the  code,  and  to  provide  sample  data  to  demonstrate  the  Com¬ 
parative  Analysis  tools.  These  are  the  "trial"  systems  referred  to  throughout  this  report.  During 
this  process,  some  implementation  errors  and  restrictions  were  discovered  in  the  trial  systems, 
as  detailed  below; 

•  Some  systems  did  not  support  the  labeCADDRESS  clause  (used  in  the  ACEC  to  measure 
code  expansion  size). 

•  Many  systems  do  not  support  tying  tasks  to  interrupts. 

•  Several  systems  do  not  support  a  type  ’SMALL  specifying  a  fixed  point  delta  which  is  NOT 
a  power  of  two. 

•  Many  systems  did  not  support  asynchronous  I/O  operations  -  that  is.  an  I/O  operation  in  any 
task  causes  the  program  to  halt  until  the  I/O  completes. 

•  Some  systems  failed  to  reclaim  implicitly  allocated  space.  The  LRM  does  NOT  require  that 
the  space  be  reused,  so  this  is  an  implementation  restriction  rather  than  an  error,  but  may  be  a 
serious  problem  to  any  program  which  needs  to  run  for  a  long  time. 

•  Some  systems  restrict  the  type  of  files  that  can  be  processed.  Test  problems  using  a  SE- 
QUENTIALJO  instantiated  with  an  unconstrained  type  (that  is,  STRING)  were  not  ac¬ 
cepted  by  many  systems,  as  were  problems  using  sequential  and  direa  files  with  a  variant 
record.  This  feature  is  not  supported  by  many  of  the  trial  systems.  On  the  DEC  Ada  system 
the  test  problems  would  not  run  using  the  default  FORM  strings  and  a  system-dependent  ad¬ 
aptation  was  necessary. 

•  The  test  problem  which  calls  on  an  assembly  language  procedure  did  not  work  on  all  sys¬ 
tems.  In  several  cases,  the  trial  systems  claim  to  support  the  facility,  but  the  provided  docu¬ 
mentation  was  unclear,  and  the  initial  anempt  to  adapt  the  test  problem  failed. 

•  Capacity  limitations  -  some  programs  were  too  large  to  be  compiled  by  some  of  the  systems. 

•  The  implementation-dependent  type  SYSTEM.  ADDRESS  is  converted  to  an  integer  type  to 
calculate  sizes.  The  size  of  this  integer  type  is  implementation  dependent  and  may  need  to  be 
adapted  for  use  on  different  systems.  The  need  for  adaptation  is  awkward. 
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•  Some  implementations  imposed  restrictions  on  length  clauses.  Several  only  supported  the 
declaration  of  specific  predefined  sizes.  For  example,  several  would  not  accept  a  specifica¬ 
tion  for  a  thice-bit'wide  field. 

•  Pre-emptive  priority  scheduling  was  not  supported  on  ail  the  trial  systems.  This  caused  some 
of  the  test  problems  to  report  a  run-time  error. 

•  One  system  (for  an  embedded  target)  did  not  support  any  file  I/O. 

•  Some  systems  did  not  support  an  option  to  specify  tasking  discipline  (time-sliced  versus  run- 
till -blocked).  This  is  NOT  required  by  the  LRM,  but  some  classes  of  programs  find  it  help¬ 
ful. 

•  On  some  systems  the  algorithm  used  to  choose  among  the  open  alternatives  in  a  SELECT 
statement  is  not  "fair."  (Some  open  alternatives  do  not  get  chosen  at  all  on  these  systems.) 
Fairness  is  NOT  required  by  the  LRM,  but  some  classes  of  programs  find  it  helpful. 

•  A  closure  (recompilation)  facility  is  required  for  some  of  the  problems  in  the  Systematic 
Compile  Speed  Group  (SY).  This  was  not  available  on  all  trial  systems. 

•  Verification  errors  were  fairly  common  on  the  Silicon  Graphics  system. 

The  problem  with  verification  errors  on  the  Silicon  Graphics  seems  to  be  largely  due  to  a 
fault  in  the  command  files.  When  the  command  file  was  corrected,  results  changed  from  438 
verification  failures  and  135  large  negative  times  to  43  verification  failures  and  no  large 
negative  times. 

Most  of  these  problems  are  not  problems  in  the  ACEC  as  much  as  they  are  restrictions  of  the 
trial  systems  discovered  by  running  the  ACEC.  They  could  be  considered  portability  problems 
if  it  were  a  goal  to  run  the  ACEC  without  modification  on  aU  validated  compilation  systems; 
however,  since  doing  so  would  require  the  ACEC  to  avoid  testing  any  system-dependent  fea¬ 
tures  and  to  code  around  errors  in  compilation  systems,  this  has  never  been  a  goal  of  the  ACEC. 

3.3  POSSIBLE  ENHANCEMENTS 

There  are  several  possible  enhancements  to  the  ACEC  product  which  would  increase  the  use¬ 
fulness  of  the  ACEC  to  users,  making  the  ACEC  easier  and  faster  to  use  (particularly  faster 
with  respect  to  programmer  time  and  effort)  and  to  increase  the  coverage  of  ACEC. 

•  Merge  ACEC  with  the  AES, 

-  Incorporate  ideas  on  user  interface  from  the  AES. 

>  Interactive  querying  of  status  of  testing. 

What  problems  ran  successfully,  with  errors,  or  were  not  tutenqrted  yet. 

>  Interactive  selection  of  sets  of  tests  to  be  run. 
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>  Using  results  of  early  testing  to  automate  later  adulations. 

Have  Pretest  query  the  user  once  as  to  what  measurement  technique  to  use. 

-  Incorporate  selected  tests  or  groups  from  AES. 

>  Adapt  existing  AES  test  problems  and  incorporate  them  into  the  ACEC. 

>  Consider  classes  of  AES  tests  and  add  problems  to  cover  the  topic. 

Test  for  implementation  dependencies. 

Collect  data  to  determine  run-time  system  size. 

-  Incorporate  assessors  for  other  environment  capabilities. 

>  Profiler  /  Timing  analysis  tool 

>  Cross  Reference  /  Interactive  browser 

>  Test  Coverage  tool 

>  Test  Bed  Generator 

>  Pretty  Printer 

>  Stub  Generator 

>  Syntax  Editor 

>  Assertion  checker 

>  Name  Expander 

>  Source  Generator 

•  Enhance  user  documentation 

There  are  several  areas  in  which  the  user  documentation  might  be  enhanced.  Some  users 
have  suggested  that  the  ACEC  documentation  be  expanded  to  include  all  issues  which  might 
affect  compiler  selection. 

A  hypertext  version  of  the  Guides  and  VDD  would  be  valuable. 

•  Enhance  the  statistical  robusmess  of  the  problem  factor  estimates  conqnited  by  CA. 

In  the  current  CA  program  when  comparing  five  (similar)  systems,  if  the  measurements  from 
four  are  close  and  the  fifth  system  is  very  much  larger,  the  calculated  mean  will  be  far  from 
all  the  systems  and  so  ALL  systems  will  be  flagged  as  outliers.  In  such  a  case,  it  would  be 
better  if  a  more  robust  estimator  of  central  location  were  used  which  would  be  closer  to  the 
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four  similar  systems  and  so  only  the  fifth  system  would  be  flagged  as  an  outlier.  While  using 
a  median,  as  in  the  Release  2.0  analysis  tool  MEDIAN,  would  not  permit  the  computation  of 
confidence  intervals,  there  are  statistical  techniques  which  can  be  used  to  compute  an  esti¬ 
mate  of  the  mean  which  is  less  affected  by  one  or  two  outliers.  CA  uses  such  a  technique 
when  calculating  system  factors. 

•  Enhance  the  statistical  robusmess  behavior  of  CA  with  respect  to  missing  data. 

Where  the  set  of  systems  being  analyzed  by  CA  contains  a  wide  performance  range  of  sys¬ 
tems  and  for  a  few  test  problems  one  of  the  very  slow  systems  has  missing  data,  the  compu¬ 
tation  of  the  mean  for  these  problems  will  be  greatly  influenced  by  the  missing  data.  The 
estimated  means  derived  by  ignoring  the  missing  data  can  be  very  different  from  the  esti¬ 
mates  which  would  have  been  made  if  the  data  from  the  slow  system  were  available.  This 
different  estimate  for  the  problem  mean  can  also  affect  the  calculation  of  residuals  and  the 
identification  of  outliers. 

The  processing  of  the  GENERIC  group  for  the  Summary  Over  AH  Groups  Report  for 
compile-time  analysis  shows  this  effea;  data  for  the  two  slower  compilation  systems  were 
missing  and  this  resulted  in  ALL  three  other  systems  being  flagged  as  very  slow  outliers. 
This  example  is  also  counter- inmitive  in  that  all  the  systems  are  shown  as  slower  than  aver¬ 
age,  which  suggests  that  the  estimate  for  the  average  should  be  reduced. 

•  Extend  coverage  of  ACEC  assessors  to  test  for  reliability/robusmess  of  a  system  by  using 
random  self-checking  program  generators  (tailorable  to  emphasize  different  language  areas). 
Observed  failure  rates  for  such  programs  can  be  used  to  estimate  the  number  of  errors  in  the 
compilation  system. 

•  Provide  more  flexible  reporting  options  for  SSA,  permitting  users  to  select  a  range  of  repon 
styles:  from  one  similar  to  the  AES  with  much  textual  descriptions  and  interpretation  of  re¬ 
sults;  to  one  similar  to  the  current  SSA  report;  tc  a  very  terse  style  listing  only  feamre  name 
and  results. 

The  last  form  may  permit  a  presentation  format  which  lists  results  ftnrn  several  different  sys¬ 
tems  on  one  report,  facilitating  comparisons  between  systems. 

•  Provide  for  easy  comparisons  of  assessor  results.  One  simple  scheme  for  this  would  be  to 
provide  for  a  one  line  list  of  question  name  and  results  from  different  systems  in  separate 
columns  on  one  page  (this  is  applicable  when  results  can  be  presented  in  a  few  columns). 

It  would  be  possible  to  perform  some  analysis  on  the  assessor  results  data,  even  when  it  is 
not  numeric  (only  reflecting  YES/NO  results)  by  highlighting  questions  where  systems  did 
not  have  the  same  response.  If  the  answers  to  a  question  are  tdl  YES  or  all  NO,  the  question 
does  not  differentiate  between  systems  and  may  be  omitted  in  a  report  designed  to  highlight 
differences  between  systems. 

•  Provide  an  assessor  for  compilation  system  reliability/robustness. 
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•  Support  for  testing  Ada  9X  feamics  and  changing  cxisimg  ACEC  code  which  turns  out  to  be 
in  conflict  witft  Ada  9X.  This  is  appUcable  after  9X  u  adopted 
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4.  SUMMARY 


Many  of  the  changes  made  in  Release  3.0  offer  help  in  several  areas: 

•  Ease  of  use  is  enhanced  by  providing  a  MENU  for  the  analysis  tools,  the  Pretest  procedure, 
and  the  renaming  to  reflect  test  purposes  which  eases  the  selection  of  subsets. 

•  Capability  is  extended  by  providing  a  Capacity  Assessor  and  additional  performance  tests. 

•  The  utility  of  the  statistical  analysis  is  enhanced  by  providing  confidence  intervals  for  system 
factors. 

Running  the  ACEC  Release  3.0  on  the  trial  systems  demonstrated  that  not  all  validated  Ada 
compilations  systems  will  accept  all  valid  Ada  programs,  even  excluding  those  with  "obvious " 
system  dependencie.s  such  as  tying  tasks  to  interrupts.  Ada  compilers  are  large  and  complex 
programs  and  it  should  not  be  too  surprising  that  they  are  not  error-free.  While  the  ACEC  was 
not  designed  to  test  correemess  of  compilers,  it  has  (since  Release  1.0)  been  effective  in  discov¬ 
ering  errors  in  implementations.  This  should  be  considered  as  a  beneficial  side-effect  of  the 
ACEC  to  the  extent  that  it  has  encouraged  compiler  implementors  to  correa  errors. 

Having  Independent  Validation  and  Verification  (FV&V)  has  been  helpful  to  the  ACEC  pro¬ 
gram.  Comments  from  experienced  beta  testers  have  permitted  enhancements  to  be  made  to  the 
product  before  general  release  by  exposing  the  product  to  individuals  with  a  different  profes¬ 
sional  perspective  and  background  than  the  developers.  Especially  with  respect  to  documenta¬ 
tion,  an  independent  reviewer  can  call  attention  to  issues  which  seem  too  obvious  to  the  devel¬ 
opers  to  require  explanation  or  where  descriptions  are  not  clear  to  individuals  not  already 
familiar  with  the  product.  ACEC  testers  have  used  different  compilation  systems  and  have 
pointed  out  issues  which  were  not  problems  on  the  trial  systems  Boeing  used. 

The  input  from  the  external  reviewers  has  resulted  in  changes  which  have  improved  the  ACEC 
product  by  enhancing  usability,  clarity  of  documentation,  accuracy  of  tests,  scope  of  coverage 
of  the  suite,  and  the  utility  of  analysis  program  results. 

While  there  remain  areas  of  possible  enhancement,  as  discussed  in  the  prior  section.  Release  3.0 
provides  a  solid  baseline  for  Ada  compilation  system  evaluation  which  is  both  easier  to  use  than 
the  prior  releases  and  provides  enhanced  functional  capabilities. 
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5.  NOTES 

This  section  contains  information  only  and  is  not  conuactually  binding. 
5.1  ABBREVIATIONS  AND  ACRONYMS 


ACEC 

Ada9X 

Ada  Compiler  Evaluation  C^ability 

Future  MIL_STD_1815B 

CA 

CPU 

Comparative  Analysis 

Central  Processing  Unit 

I/O 

Input  /  Output 

LRM 

(Ada)  Language  Reference  Manual  (MIL-STD-1815A) 

SSA 

Single  System  Analysis 

TOR 

Technical  Operating  Report 

VDD 

Version  Description  Document 
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